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The procedure presented in this paper has been developed to determine 
in a post hoc manner, if it is likely that the data from a repeated measures 
experiment, have met the assumption of additivity for a univariate analysis 
of variance model. Of particular concern was the type of non-additivity which 
could arise, if a component of the measures taken was some unknown function 
of one or more unknown individual difference variables and one or more of the 
independent variables taken into consideration in the experimental design on 
which the analysis was based. An individual difference variable being some 
variable with a finite number of levels to which subjects or individuals can 
be assigned. The procedure will be developed and applied to a set of data. 

The importance of determining if it is likely that there is an individual 
difference variable of which the treatments are a function is demonstrated in 
an example referred to by Jensen (1967) of a study by Hovland (1939) who per- 
formed an experiment in which no statistically significant differences were 
found between massed practice and distributed practice on paired associate 
learning tasks. Upon subsequent examination of the data, however, it was deter- 
mined that 44% of the subjects improved their performance more rapidly on distri- 
buted practice and 38% learned faster with massed practice, inus it is likely 
that had an additional factor which reflected the individual differences suggested 
by the previous percentages been Included in the design of the experiment, a 
significant individual difference factor by massed vs. distributed practice in- 
teraction would have been found. 

Figure 1 is an example of a repeated measures design. Table 1 is its con- 
comitant summary table for a univariate analysis of variance, specif ing the 
appropriate sources of varla tion, degrees of freedom, expected mean squares and 



F ratios under the assumptions of normality, Independence, homoscedastlclty 
for treatment groups, additivity of treatment effects as well as equal 
variances and equal covariances In the residual variance - covariance matrix 
for repeated measures. 

The repeated measures design Is of particular importance to those who 
are interested in doing experiments concerned with learning. For learning 
studies repeated trials on a single type of task, repeated trials on different 
tasks, or both together crossed with and following different treatments are 
not at all uncommon. The design provides for not only efficient use of hard to 
obtain subjects but as well for a means of investigating important theoretical 
questions involving the interaction of treatments with tasks over time. 

Table 2 is an example of univariate analysis of variance summary data 
obtained from a learning experiment which fits the design in Figure 1, where 
A and B are treatment factors with two levels each and R represents three kinds 
of tasks. The statistics calculated indicate significant B, R, RA, and RB effects 
at the a*. 05 level. The critical values of F were based on degrees of freedom 
adjusted by an estimate of c, where e is a function of the variance-covariance 
matrix for repeated measures, e^FC^). The estimate of e was obtained by sub- 
stituting the sample variance-covariance matrix in the argument in place of ijl, 
e“F(|). (See Theorem 6.1, Box (1949, 1950) ). Because there is a possibility 
that the data do not fit the additive model assumed in doing the analysis of 
variance and because of the consequences of such a violation it is suggested 
tile assumption be tested. One approach to testing the hypothesis of additivity 
would be to employ the Tukey 1 degree of freedom test for non- additivity (ScheffA, 
1959) which poses as an alternative to the additive model, a multiplicative model. 
It is suggested, however, that a non-multlpllcatlve alternative may be present. 
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Further no extension of the Tukey test to a repeated measures situation could 
be found. Multiplicative non-additivity can be represented by an ordinal 
interaction as in Figure 2, This type of non-additivity can be eliminated 
by means of a non-linear monotonlc transformation upon the data which gave 
rise to it. The type of non-additivity which is indicated by a dleordlnal 
Interaction e.g. Figure 3, however, cannot be eliminated in this manner. The 
possible interaction in the Hovland data suggests dlsordlnallty . 

It is often difficult for an investigator to include all relevant inde- 
pendent variables in the design of his experiment. That is they may be of 
unique importance to the type of experiment being undertaken and unknown prior 
to conducting the experiment. Thus it is possible in a given experiment for 
some unknown variable, which has not been included in the design on which the 
analysis is based, to exist such that it Interacts with one or more of the 
Independent variables with which it is crossed. If such an unknown variable, 
say U, were to exist it would be one that would classify subjects in some 
manner and since subjects are crossed with repeated measures, a U x R inter- 
action would be expressed in a repeated measures by subjects nested within A 
and B (RxS:AB) Interaction, the existence of which Implies a violation of the 
assumption of additivity. 

Note that if U had as many levels as the experiment had subjects a sub- 
sequent experiment could not be designed which took U into account so as to 
eliminate the RxS interaction, for the U would be perfectly confounded with sub- 
jects. It will be assumed that this is not the case. 

It should be apparent by now that if an appropriate denominator for an F 
ratio with mean square RxS:AB in the numerator can be found, we shall have a 
test for the types of non-additivity with which we are concerned. Inspection of 
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the expected value of mean square RxS:AB indicates that the desired denomi- 
nator should be a mean square which provides an unbiased estimate of within 
cell variation. The problem of finding that denominator took two different 
approaches which fortunately converged upon the same solution. 

In the first approach it was noted that because the design which repre- 
sents subjects crossed with repeated measures (Figure 4) has only one obser- 
vation per cell, it has no degrees of freedom for an estimate of within cell 
variation. However, it was also noted that the within cell variance corre- 
sponds to what is referred to in classical measurement theory as the variance 
error of measurement. Then if the classical measurement theoretic approach 
is taken which assumes that an observation X is composed of a "true" score T 
and an independent "error" score e, X » T + e from which it follows that 

the observed score variance is equal to the sum of the true score variance 

2,4 2 

Off and the error variance o^ , 

ar ^2 X q2 

X T e 

then, if we analogously consider an observed value, , as the sum of "true" 



and "error" parts we have, 

\SAB ■“ ^RSAB ®RSAB * 

1 

Therefore, the variance for the source RxS:AB (o^ ) which is estimated by the 

fa • aK •' 



mean square RxSrAB can be partitioned into a "true" interaction part. 





and an uncorrelated "error" part, denoted a^, we would have 



e 



the following equality. 




1 



The X will be left out of the subscripts for interaction terms. Thus for 
the source RxS:AB, the corresponding mean square* 



l 



Again from classical measurement theory 
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^xx^ * 



2 o 2 

then the estimate of o , + o^ Is MS„o and the estimate of o Is 

rs:ab(T) ® RS;AB e 

2 

"■ ^xx^* estimates are distributed as Independent Xa» 



then 
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RS:AB 



1 ~ r 
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given that » 0, will be distributed as central F and there is a 

\ i ^ 

test for the source of variance RxSsAB. 

The second approach involved considering the values assigned to each sub- 
ject as a composite of constltuate parts of items either crossed with or nested 
within the repeated measures. The items were considered as a random sample 
from some population of Items and thus as a random independent variable. Adding 
Items as nested to the design we arrived at what is represented in Figure 5. 

Table 3 is it's concomitant summary table. By Inspection of expected mean squares 
the source for the Items by subjects nested within A and B and R interaction 
(IxSiABR) provides the denominator for the F ratio to test the RxS:AB interaction. 

It can be shown most clearly how the two approaches converge if the Hoyt 
analysis of variance method of estimating r is employed. In this case 

- MSgubjecte ~ »Ssub » sterns 
^®aubjects 



or In our case to remove treatment effects 



■ ”^RS!AB ~ ”^IRS;AB 
“®RS:AB 

with Items considered as crossed, 

^^RS:AB “ ^^IS;ABR 
^^RS:AB 

with items considered as nested as in Table 3. 
Then consider only the nested case 
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which is identical to the F test statistic obtained from the design in Figure 2. 

MS 



Similarly for the crossed case 

F “ 



RS:AB 



MS. 



ISR:AB 

Thus if there is no RxSsAB effect the two ratios would be distributed as 
central F's^ with respectively e(r~l) (s-l)g, e(l-l) (r-1) (s-l)g and e(r-l) 
(s-l)g, e(l-l) (s-l)rg degrees of freedom for the crossed and nested cases. Where 
G is a function of the variance-covariance matrix as previously noted. 



An article by Ilsu and Feldt (1970) and two dissertations (Hsu, 1968) and 
(Lunney, 1968) indicate that an analysis of variance can be used on binary 
data with 20 or more degrees of freedom for error if P the probability of 
a "one” is between .2 and .8 or with mope than 40 d.f. error where .1<P<.9* 
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Returning to the demonstration data the initial 2x2x3 design was 
augmented by the inclusion of the random independent variable items to a 
2 X 2 X 3 X 4 design. The summary data in Table 4 Indicate that for the 
2 X 2 X 3 X 4 design there is a significant RxS:AB effect. Now i'. can be 
told that another independent variable C(sex) had been included in the 
original design but Ignored for purposes of demonstrating our technique. 

Table 5 is the summary data for the analysis on the 2 x 2 x 2 x 3 design 
which includes C. Note the significant BxCxR interaction which would have 
been predicted from the RxS;AB interaction if U^C. But such a convenient 
conclusion would be unrealistic and Table 6 shows that if items as a random 
source is again included there is still a significant RxS:ABC effect. Having 
no further "put up" situations to draw candidates for a U from, the RxS in- 
teractions were plotted and five main types of response curves were detected 
by ocular inspection. The response curves are graphed in Figure 6. 

In conclusion a strategy is suggested for those who plan to do experi- 
ments of the type discussed and who are not confidant that they have all rele- 
vant independent variables in mind. The suggestion is to run a pilot study and 
look for non-additivity. If it is found, try to categorize the subjects and 
then interview them and take other measures in an attempt to determine what 
occured. Then design the next experiment taking into consideration what was 
discovered in the pilot. 



TABLE 1 

Analysis of Variance Summary Statistics and Parameters 
for a 2 X 2 X 3 Repeated Measures Design 



Degrees of Freedom 


Expected Mean Squares 


F Ratios 


(a-1) 


0^ + 0^ . + 


MS^ 

^%:AB 


(b-1) 


+ of , + o2 
e s : ab b 


MSb 

:AB 


(a-l)(b~l) 


°e + °8:ab + "ib 


«®AB 

”®S:AB 


(a-l)ab 


K 

e s sab 




(r-1) 


of + 0^ , + 0^ 

e rssab r 






”®RS :AB 


(r-1) (a-1) 


“e “rsiab + °?a 


«Ska 




^^RSsAB 


(r-1) (b-1) 


®fs:ab ®fb 


MSrb I 




, »SrS:AB j 


(r-1) (a-1) (b-1) 


“e + ■’ratab + °rab 


«Srab i 




^^RSsAB 1 


(r-1) (s-l)ab 


*^rs:ab 


i 



( Notes The coefficients of the variance components have been omitted. ) 



TABLE 2 



Analysis of Variance Summary Data for a 2 x 2 x 3 Repeated Measures Design 




Source Degrees of Freedom Mean Squares F Statistics Critical Values of F^ 



A 


1 


5.01 


1.93 


B 


1 


57.42 


22.17* 


AB 


1 


0.13 


.05 


S;AB 


60 


2.59 




R 


2 


40.72 


57.35* 


RA 


2 


5.51 


7.76* 


RB 


2 


5.82 


8.20* 


RAB 


2 


0.26 


.37 


RS:AB 


120 


.71 





4.00 

4.00 

4.00 

3.09 

3.09 

3.09 

3.09 



^Value at a-. 05. degrees of freedom adjusted by the Box function of 
covariance matrix. 
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Fig. 3 An example of a diaordlnal Intejcaction. 
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Fig. 5 Experimental design for two independent variables and four items nested within 
three repeated measures. 
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TABLE 3 



Analysis of Variance Sumnary Statistics and Parameters 
£ora2x2x3x4 Repeated Measures Design 



Source 

! 


Degrees of Freedom 


Expected Mean Squares 


F Ratios 


A 


(a-1) 


°ls:abr ‘^s:ab ‘^altr ''' 




B 

1 


(b-1) 


“istabr * “Jiab “blir “b 




AB 


(a-1) (b-1) 


®ls:abr ®s:ab ®abl:r ®ab 




S:AB 


(s-D(b-l) 


2 2 
Is : abr % : ab 




R 


(r-1) 


‘’larabr + ®ra:ab + “Lr + 




I:R 


(l-l)r 


°ls:abr ^l:r 

ls:abr rs:ab ®al:r ^ ®ar 


"Si:R 


AR 


(a-D(r-l) 


"®IS:ABR 


AI:R 

BR 


(a-D(l-l)r 
(b-1) (r-1) 


oj + j 

Is: abr al:r 

°ls:abr ®rs:ab ^bl:r''’ ®br 


”SaI:R 

IS: ABR 


BI:K 


(b-1) (l-l)r 


oj K + 0 I 4 
Is: abr bl:r 

°ls:abr ®rs:ab °'abl:r ®abr 


”^BI:R 


ABR 


(a-1) (b-1) (r-1) 


MS 

IS: ABR 


ABi:R 

) 

1 


(a-l)(b-l)(l-l)R 


0 ? , . + 0 ^ . . 

Is: abr abl:r 


^®ABI:R 
IS: ABR 


i RS:AB 


(r-l)(s-l)ab 


K + u 

Is: abr rs:ab 

0 ^ 

Is: abr 


”®RS:AB 


IS:AHR 


(l-l)(s-l)abr 


”^1S:ABR 



( Note: The coefficients of the variance components have been omitted. ) 
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TABLE A 



Analysis of Variance Summary Data for a2x2x3x4 Repeated Measures Design 



Source 


Degrees of Freedom 


Mean Squares 


F Statistics Critical Values of F^ 


A 


1 


1.25 




B 


1 


14.35 




AB 


1 


.03 




S:AB 


60 


.64 




R 


2 


10.18 




1:R 


9 


1.16 


6.56* 2.10 


RA 


2 


1.38 




AI:R 


9 


0.18 


1.06 2.10 


RB 


2 


1.46 




BI;R 


9 


0.28 


1.64 2.10 


RAB 


2 


0.06 




ABI:R 


9 


0.16 


.94 2.10 


RS:AB 


120 


0.17 


1.54* 1.28 


IS:ABR 


540 


0.11 





^ Value at a-. 05, degrees of freedom adjusted by the Box function of the variance*" 
covariance matrix. 
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TABLE 5 

Analysis of Variance Summary Data fora2x2x2x3 Repeated Measures Design 




Source Degrees of Freedom Mean Squares F Statistics Critical Values of F^ 



A 1 

B 1 

C 1 

AB 1 

AC 1 

BC 1 

ABC 1 

S:ABC 56 

R 2 

RA 2 

RB 2 

RC 2 

RAB 2 

RAC 2 

RBC 2 

RHBC 2 

RS:ABC 112 



5.01 


1.88 


57. A2 


21.51* 


.88 


.33 


.13 


.05 


1.17 


.AA 


2.29 


.86 


1.17 


. AA 


2.67 




AO. 72 


58.88* 


5.51 


8.10* 


5.83 


8.57* 


l.OA 


1.53 


.26 


.38 


.30 


.AA 


2.12 


3.12* 


.A8 


.73 


.68 





A. 00 
A. 00 
A. 00 
A. 00 
A. 00 
A. 00 
A. 00 

3.10 

3.10 

3.10 

3.10 

3.10 

3.10 

3.10 

3.10 



coiarlMcrmatilx!*"*® adjusted by the Box function of the variance- 




TABLE 6 



Analysis of Variance Summary Data fora2x2x2x3x4 Repeated Measures Design 



Source 

f 


Degress of Freedom Mean Squares 


F Statistics 


Critical Values of F^ 


A 


1 


1.25 






B 


1 


14.36 






C 


1 


.22 






AB 


1 


.03 






AC 


1 


.29 






BC 


1 


.57 






ABC 


1 


.29 






S:ABC 


56 


.67 






R 


2 


10.18 






I;R 


9 


1.16 


6.82* 


2.11 


AR 


2 


1.38 




AI:R 


9 


.18 


1.05 


2.11 


BR 


2 


1.46 




BI:R 


9 


.28 


1.64 


2.11 


CR 


2 


.26 




CI;R 


9 


.10 


.58 


2.11 


ABR 


2 


.06 




ABI:R 


9 


.16 


.94 


2.11 


ACR 


2 


.07 




ACI;R 


9 


.21 


1.23 


2.11 


BCR 


2 


.53 




BCI:R 


9 


.11 


.65 


2.11 


ABCR 


2 


.12 




ABCI:R 


9 


.13 


.76 


2.11 


RS:ABC 


112 


.17 


1.42* 


1.29 


IS: ABCR 


50A 


.12 





Value at a-. 05, degrees of freedom adjusted by the Box function of the variance- 
covariance matrix i 











Fig. 6 Some response curves from Che demons t rift t Ion data. 



References 






Box, G. E. P, Some theorems on quadratic forms applied in the study of 
analysis of variance problems: I. Effect of inequality of variance 

in the one-way classification. "Ann, Math. Statist.," 1954, 25. 

290-302. — 

Box, G. E. P. Some theorems on quadratic forms applied in the study of 

analysis of variance problems: II, Effects of inequality of variance 

and of correlation between errors in the two-way classification. 

"Ann. Math. Statist. ,"1954, 2 ^, 484-498. 

Hovland, C. I. Experimental studies in rote-learning theory. V, Comparison 
of distribution of practice in serial and paired-associate learning. 
"Journal Exp, Psychol, ,"1939, 622-633. 

Hsu, T. -Chi and Feldt,’ Leonard S. The effect of limitations on the number 
of criterion score values on the significance level of the F~test. 
"American Educational Research Journal," 1969, (6, No. 4, 515-528. 

Jensen, A, R. "Varieties of individual differences in learning." In R, M. 
Gang& (Ed.,) Learning and Individual Differences . New York: Merrll, 

1967, 265pp. 

Lunney, Gerald Hugh A monte carlo investigation of basic analysis of variance 
models when the dependent variable is a bernoulll variable. University 
of Minnesota, 1968, 

Scheff&, H. "The analysis of variance," John Wiley, Nev; York, 1959. 




